Skip to content

feat(ci): opt-in answer-quality/targeting eval on canary dispatch (Phase E instrument)#1013

Merged
BigSimmo merged 2 commits into
mainfrom
claude/clinical-kb-pwa-review-asi3wb
Jul 20, 2026
Merged

feat(ci): opt-in answer-quality/targeting eval on canary dispatch (Phase E instrument)#1013
BigSimmo merged 2 commits into
mainfrom
claude/clinical-kb-pwa-review-asi3wb

Conversation

@BigSimmo

@BigSimmo BigSimmo commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Summary

First Phase E delivery (user-approved with budget): give the canary an opt-in answer_quality_eval dispatch input that runs the 30-case eval:answer-quality fixture — five answer-quality metrics (relevance, readability, artifact leaks, intent coverage, fail-closed) plus the per-intent structural targeting metric (dose→figure, red-result→withhold action, monitoring→schedule). E-1 recon found this instrument has never run in CI, and the canary's answer gate samples only 8 of the 44 eval:quality cases — so the Phase E baseline needs this step.

Design constraints honored:

  • Dispatch-only and default-off: scheduled runs and every existing dispatch shape are byte-identical (if: workflow_dispatch && input == 'true'); follows the feat(ci): eval-canary dispatch inputs — staged ranking-config override + branch evals (ADDENDUM 4 B-PR-1 / D-1) #1003 input-only precedent.
  • Informational by construction: the script exits 0 without --targeting-floor, so this step can never fail a canary — quality gates stay owned by the eval:quality step.
  • The tee'd log lands in .local/eval-canary/ and rides the existing pinned artifact upload; 45-minute job timeout comfortably absorbs the ~10-minute step.

Next (after merge): the E-2 baseline dispatch — answer_case_limit=44 + answer_quality_eval=true — one run giving the full answer-quality baseline on both instruments plus the golden retrieval eval as the built-in no-regression net. Estimated $3–8 of the user-authorized ≤$20 Phase E envelope; reported before/after dispatch.

RAG impact: no retrieval behaviour change — dispatch-input-only workflow addition; retrieval steps, schedule, and defaults untouched.

Verification

  • npm run check:github-actions — pass (no new actions; existing pins untouched)
  • npm run check:ci-scope — pass
  • npm run check:gate-manifest — pass
  • npx prettier --check on both changed files — clean
  • No provider calls in this PR (workflow-definition change only); the paid dispatch happens separately under the approved envelope

Risk and rollout

  • Risk: low — a conditioned, default-off step in a dispatch-only path; scheduled behaviour provably unchanged (condition requires workflow_dispatch AND explicit 'true'). The step is informational and cannot fail the run.
  • Rollback: revert.
  • Provider or production effects: none from this diff. When the input is enabled on a dispatch, the step spends 30 live generation cases ($1–3) — always under an explicit user-approved dispatch.

Clinical Governance Preflight

  • Source-backed claims still require linked source verification before clinical use — no behaviour change
  • No patient-identifiable document workflow was introduced or expanded
  • Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — same env block, no new secret exposure
  • Service-role keys and private document access remain server-only
  • Demo/synthetic content remains clearly separated from real clinical sources
  • Source metadata, review status, and outdated/unknown-source behavior remain conservative
  • Deployment classification/TGA SaMD impact was checked: no clinical decision-support behaviour changed — measurement only

🤖 Generated with Claude Code

https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9


Generated by Claude Code

Summary by CodeRabbit

  • New Features

    • Added an optional dispatch-only workflow input to run an additional 30-case answer-quality/targeting evaluation.
    • When enabled, the workflow produces a dedicated log artifact containing the targeting evaluation output.
  • Documentation

    • Updated the Branch Review Ledger with the Phase E kickoff details, including the default-off behavior, canary sampling notes, and baseline follow-up parameters.

…ase E instrument)

Adds an answer_quality_eval dispatch input to eval-canary that runs the
30-case eval:answer-quality fixture (five quality metrics + per-intent
structural targeting) after the eval:quality gate subset. The instrument
existed but was never run in CI. Dispatch-only and default-off: scheduled
runs and existing dispatch shapes are byte-identical, and the step is
informational by design (exit 0 without --targeting-floor) so quality gates
stay owned by eval:quality. The tee'd log rides the existing artifact upload.
Follows the #1003 input-only precedent; retrieval steps untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
@supabase

supabase Bot commented Jul 20, 2026

Copy link
Copy Markdown

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.


Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

@BigSimmo
BigSimmo marked this pull request as ready for review July 20, 2026 22:44
@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: c747c86e-975a-4951-84c8-c9d411cba307

📥 Commits

Reviewing files that changed from the base of the PR and between f2fada0 and 22d9262.

📒 Files selected for processing (1)
  • docs/branch-review-ledger.md
🚧 Files skipped from review as they are similar to previous changes (1)
  • docs/branch-review-ledger.md

📝 Walkthrough

Walkthrough

The canary workflow adds an optional dispatch input for running the answer-quality evaluation and writing a dedicated log. Scheduled and default dispatch behavior remains unchanged, and the review ledger documents the update.

Changes

Answer-quality canary evaluation

Layer / File(s) Summary
Add optional answer-quality evaluation
.github/workflows/eval-canary.yml, docs/branch-review-ledger.md
The workflow adds the default-off answer_quality_eval dispatch input and conditionally runs npm run eval:answer-quality; the ledger records the dispatch-only behavior and evaluation details.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • BigSimmo/Database#624: Updates the answer-quality evaluation’s latency context and thresholds in the related evaluation flow.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the opt-in canary answer-quality eval addition.
Description check ✅ Passed The description covers the required summary, verification, risk, and governance sections and is mostly complete.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/clinical-kb-pwa-review-asi3wb

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@BigSimmo
BigSimmo enabled auto-merge (squash) July 20, 2026 22:44
…a-review-asi3wb

# Conflicts:
#	docs/branch-review-ledger.md
@BigSimmo
BigSimmo merged commit 5b4098d into main Jul 20, 2026
18 checks passed
@BigSimmo
BigSimmo deleted the claude/clinical-kb-pwa-review-asi3wb branch July 20, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants